1. Technical Field
The invention is related to texture map construction, and more particularly to a system and process for generating an optimal texture map of a scene from a plurality of textures each of which is reconstructed from multiple input textures representing the same portion of the scene and taken from images of the scene captured from different perspectives.
2. Background Art
Texture mapping is an established rendering technique used to enhance the realism of 3D models. In computer vision, 3D models are typically constructed using multiple images (and possibly range data). Their textures are also recovered using combinations of appropriately extracted parts of the source images. Currently, textures or images are manipulated (e.g., warped) using techniques that are simplistic approximations of the true mapping function, which results in suboptimal appearances in the recovered textures. These approximations are used primarily because of their simplicity in implementation or limitations of hardware.
Thus, an important issue is how these textures can be extracted as accurately as possible from multiple views. Assuming that all surfaces are Lambertian, a final texture is typically computed as a linear combination of the reference textures. This is, however, not the optimal means for reconstructing textures, since this does not model the anisotropy in the texture projection. Furthermore, the spatial image sampling may be quite variable within a foreshortened surface.
Generating an optimal texture map not only has implications on improving realism of the recovered 3D model. It can also apply to computer vision techniques that rely on analysis by synthesis. Such computer vision techniques reconstruct intermediate appearances for comparison with input images in order to refine the desired output. A typical example is the direct recovery of 3D geometry and texture from multiple reference images [6]. In another, Morris and Kanade [13] find the best triangulation for a given set of feature point correspondences across multiple images. The metric used is the reprojection error for a given hypothesized triangulation. Generation of correct textures is critical for such techniques.
There has also been a significant amount of work done on generating an image with a resolution higher than its individual sources, i.e., super-resolution. This can also be considered as recovering an optimal texture map from multiple (smaller resolution) texture maps seen at different views. Thus, generation of accurate textures is critical for these techniques as well. Current super-resolution approaches can be categorized as being interpolation-based [8, 17, 9], frequency-based [18, 10, 11], or reprojection-based [2, 16]. While producing acceptable results, the introduction of even more accurate and efficiently computed texture maps would be welcomed.
The generation of optimal textures is also critical for the increasingly popular image-based rendering technique (IBR) of view-dependent texture mapping (VDTM) [3]. There is typically photometric variation across the views used to construct textures due to lighting changes and non-Lambertian surfaces. View-dependent texture mapping has been proposed as an image-based means of modeling photometric variation, thus enhancing realism [3]. For a given view, reference textures are typically blended based on viewpoint proximity to the corresponding reference views (in the form of a sphere view map). Others that use the sphere view map as well include [4, 14, 15]. In the “Unstructured Lumigraph” work [1], global weights for each face texture are computed based on ray angular difference, estimates of undersampling, and field of view. Here again, methods for producing accurate and efficiently computed texture maps could be quite useful.
It is noted that in the preceding paragraphs, as well as in the remainder of this specification, the description refers to various individual publications identified by a numeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, “reference [1]” or simply “[1]”. Multiple references will be identified by a pair of brackets containing more than one designator, for example, [2, 3]. A listing of references including the publications corresponding to each designator can be found at the end of the Detailed Description section.